Image adjustment system, image adjustor, and image adjustment method

ABSTRACT

An image adjustment system includes a camera, an image adjustor, an image display, and a controller. The image adjustor includes an image generator and an image processor. The image generator generates a spherical image and a composite image to be synthesized with a shooting image imaged by the camera. The image processor acquires the spherical image generated by the image generator and displays the spherical image on an image display. The image processor rotates the spherical image based on instruction information output from the controller, adjusts the shooting image displayed on the image display in accordance with rotating the spherical image, adjusts the composite image in accordance with the adjusted shooting image, and synthesizes the adjusted composite image with the adjusted shooting image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of International ApplicationNo. PCT/JP2020/004700, filed on Feb. 7, 2020, and based upon and claimsthe benefit of priority from Japanese Patent Applications No.2019-025762, No. 2019-025763, No. 2019-025765, and No. 2019-025769 filedon Feb. 15, 2019, the entire contents of which are incorporated hereinby reference.

TECHNICAL FIELD

The present disclosure relates to an image adjustment system, an imageadjustor, and an image adjustment method.

BACKGROUND

Recently, a head-mounted display has attracted attention as an imagedisplay. The head-mounted display displays an image in a state ofmounted on a head of a user, so that the user can obtain a sense ofentering a virtual space (immersion). A conventional head mount displaydescribed in Patent Document 1 (JP 2005 -056295A) can display an imageimaged by an external camera through a neck work.

SUMMARY

However, in the conventional head-mounted display, it may be difficultto detect the horizontal direction of an image depending on thedisplayed image, and the horizontal direction may be erroneouslydetected. In addition, the horizontal or zenith designation may beshifted due to problems such as system synchronization. If thehorizontal direction of the image is incorrectly detected or thehorizontal or zenith designation is deviated, the user feelsuncomfortable because the displayed image does not match the user'ssense of gravity.

An object of the present application is to provide an image adjustmentsystem, an image adjustor, and an image adjustment method, capable ofeasily correcting the horizontal or zenith of an image when thehorizontal direction of the image is erroneously detected or thehorizontal or zenith designation is deviated.

An image adjustment system according to an embodiment includes: acamera; an image adjustor configured to adjust a shooting image imagedby the camera; an image display configured to display the shooting imageadjusted by the image adjustor; and a controller configured to outputinstruction information to the image adjustor. The image adjustorincludes an image generator configured to generate a spherical image anda composite image to be synthesized with the shooting image, and animage processor configured to acquire the spherical image generated bythe image generator based on the instruction information and display thespherical image on the image display, rotate the spherical image basedon the instruction information, adjust the shooting image displayed onthe image display in accordance with rotating the spherical image,adjust the composite image in accordance with the adjusted shootingimage, and synthesize the adjusted composite image with the adjustedshooting image. The camera is an omnidirectional camera configured toimage a range of 360 degrees. The image display is a head-mounteddisplay capable of mounting on a head of a user. The controller is aglove type controller capable of attaching on a hand of the user. Whenthe user watches the spherical image displayed on the image display in astate where the image display is mounted on the head of the user, thespherical image is a virtual image arranged around the user and theimage display and set to display within a range where the hand or afinger of the user is reached to the spherical image.

With the image adjustment system according to the embodiment, when thehorizontal direction of the image is erroneously detected or thehorizontal or zenith designation is deviated, the horizontal or zenithof the image can be easily corrected.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an image adjustment systemaccording to a first and second embodiments.

FIG. 2 illustrates a relationship between a spherical image and a user.

FIG. 3 is a flowchart illustrating an example of an image adjustmentmethod according to the first embodiment.

FIG. 4 is a flowchart illustrating an example of the image adjustmentmethod according to the first embodiment.

FIG. 5 is a view illustrating a relationship between a horizontal planeof a shooting image and a horizontal plane of a composite image.

FIG. 6A is a flowchart illustrating an example of an image adjustmentmethod according to a second embodiment.

FIG. 6B is a flowchart illustrating an example of the image adjustmentmethod according to the second embodiment.

FIG. 7 is a flowchart illustrating an example of the image adjustmentmethod according to the second embodiment.

FIG. 8 is a view illustrating a relationship between the horizontalplane of the shooting image and the horizontal plane of the compositeimage.

FIG. 9 is a block diagram illustrating an image adjustment systemaccording to a third embodiment.

FIG. 10 is a diagram illustrating an example of a positionalrelationship between a user and a composite image corresponding toanother user when the user is watching an image display.

FIG. 11 is a diagram illustrating an example of a positionalrelationship between a user and a composite image corresponding toanother user when the user is watching an image display.

FIG. 12 is a flowchart illustrating an example of an image adjustmentmethod according to the third embodiment.

FIG. 13 is a flowchart illustrating an example of the image adjustmentmethod according to the third embodiment.

FIG. 14 is a block diagram illustrating an image adjustment systemaccording to a fourth embodiment.

FIG. 15 is a flowchart illustrating an example of the image adjustmentmethod according to the fourth embodiment.

FIG. 16 is a flowchart illustrating an example of the image adjustmentmethod according to the fourth embodiment.

FIG. 17 is a diagram illustrating an example of a positionalrelationship between a plurality of cameras and a composite imagecorresponding to another user.

DETAILED DESCRIPTION First Embodiment

With reference to FIG. 1, a configuration example of an image adjustmentsystem 101 according to a first embodiment will be described. The imageadjustment system 101 according to the first embodiment includes acamera 102, a first communicator 103, an image display 104, a controller105, an image adjustor 110, and a server 106. The image adjustor 110includes an image processor 111, an image generator 112, and a secondcommunicator 113.

The camera 102 is an omnidirectional camera (360 degree camera) capableof imaging a range of 360 degrees. The first communicator 103 and thesecond communicator 113 are connected through a network. The imageadjustor 110 can acquire a shooting image IM1 imaged by the camera 102through the first communicator 103, the second communicator 113, and thenetwork.

Computer equipment may be used as the image adjustor 110. A CPU may beused as the image processor 111 and the image generator 112. The imagegenerator 112 and the second communicator 113 may be provided outsidethe image adjustor 110.

The server 106 is connected through the network and the firstcommunicator 103 to the camera 102, and connected through the networkand the second communicator 113 to the image adjustor 110. The server106 may acquire the shooting image IM1 imaged by the camera 102 via thefirst communicator 103 and the network, and the image adjustor 110 mayacquire the shooting image IM1 from the server 106 via the network andthe second communicator 113.

The shooting image IM1 acquired by the image adjustor 110 is input tothe image processor 111. The image processor 111 analyzes the shootingimage IM1 to recognize the horizontal direction of the shooting imageIM1. The image processor 111 may recognize the vertical direction of theshooting image IM1 or may recognize the horizontal and verticaldirections of the shooting image IM1 by analyzing the shooting imageIM1. The image adjustor 110 corrects distortion of the shooting imageIM1, executes image processing such as adjustment of the horizontalposition of the shooting image IM1, and outputs the image-processedshooting image IM1 to the image display 104.

The server 106 may correct distortion of the shooting image IM1 imagedby the camera 102, execute image processing such as adjusting thehorizontal position of the shooting image IM1, and output theimage-processed shooting image IM1 to the image adjustor 110. The imagedisplay 104 displays the shooting image IM1 subjected to imageprocessing by the image adjustor 110 or the server 106.

The image display 104 is, for example, a head-mounted display. Thecontroller 105 is, for example, a glove type controller used for VR(Virtual Reality) or the like.

FIG. 2 schematically illustrates a state in which the image display 104is mounted on the head of the user US and the controller 105 is mountedon the hands of the user US. The symbol ZE in FIG. 2 indicates thezenith. It is desirable that the zenith of the camera 102 coincides withthe zenith of the user US. By attaching the image display 104 to thehead of the user US, the user US can watch the shooting image IM1subjected to image processing by the image adjustor 110 or the server106.

The image display 104 generates attitude information PN1 on the basis ofa direction in which the user US faces and a state such as the attitudeof the user US in a state of being mounted on the head of the user US.The image processor 111 acquires attitude information PN1 from the imagedisplay 104. That is, the image processor 111 acquires the attitudeinformation PN1 based on the attitude of the image display 104. Theimage processor 111 displays on the image display 104, on the basis ofthe attitude information PN1, an image of an area corresponding to thedirection in which the user US faces and the state of the attitude ofthe user US from the shooting image IM1 imaged by the camera 102.

The controller 105 generates instruction information NN1 based on astate such as the movement or attitude of the hand of the user US or themovement or attitude of the finger of the user US in a state of beingmounted on the hand of the user US. Hereinafter, the hand or finger issimply abbreviated as the hand. An image processor 111 acquires theinstruction information NN1 from a controller 105. The image processor111 can change or adjust the shooting image IM1 displayed on the imagedisplay 104 based on the instruction information NN1.

An image generator 112 generates a composite image CM1 which is a CG(Computer Graphics) to be synthesized with the shooting image IM1. Thecomposite image CM1 is an image of a character such as an avatar. Theimage generator 112 stores the composite image CM1 in a built-in memoryor an external memory. The image processor 111 may acquire the compositeimage CM1 generated by the image generator 112 and output it to theserver 106 via the second communicator 113 and the network.

The server 106 stores the composite image CM1 in a built-in memory or anexternal memory in association with the image display 104 or the userUS. The image processor 111 acquires a composite image CM1 from theimage generator 112 or the server 106 on the basis of instructioninformation NN1, and synthesizes the composite image CM1 with a shootingimage IM1 displayed on the image display 104. Therefore, the imagedisplay 104 can display the shooting image IM1 imaged by the camera 102and the composite image CM1 acquired from the image generator 112 or theserver 106 and synthesized with the shooting image IM1.

The image generator 112 generates a spherical image VSS1 which is avirtual image constituted of a spherical surface which is CG. The imagegenerator 112 stores the spherical image VSS1 in a built-in memory or anexternal memory. On the basis of the instruction information NN1, theimage processor 111 acquires the spherical image VSS1 generated by theimage generator 112 and displays the spherical image VSS1 on the imagedisplay 104.

FIG. 2 schematically illustrates an image of the user US when the userUS watches the spherical image VSS1 displayed on the image display 104while the image display 104 is mounted on the head of the user US.

While the image display 104 is mounted on the head of the user US, whenthe user US watches the spherical image VSS1 displayed on the imagedisplay 104, the spherical image VSS1 is arranged around the user US andthe image display 104 and is set to display within a range where thehand of the user US is reached to the spherical image VSS1. The user USfeels that the hand of the user US is in contact with the sphericalimage VSS1 by moving the hand on which the controller 105 is mounted toa position corresponding to the spherical image VSS1 displayed on theimage display 104.

The controller 105 may have an actuator placed in contact with the handof the user US. On the basis of the instruction information NN1, theimage processor 111 operates the actuator when it is determined that thehand of the user US has moved to a position corresponding to thespherical image VSS1. When the actuator applies pressure to the hand ofthe user US, the user US can actually feel the feeling that the hand isin contact with the spherical image VSS1.

In a state where the spherical image VSS1 is displayed on the imagedisplay 104, when the user US moves the hand on which the controller 105is mounted in an arbitrary direction, the image processor 111 performsimage processing based on the instruction information NN1 so that thespherical image VSS1 and the shooting image IM1 displayed on the imagedisplay 104 move in accordance with the moving direction, the movingspeed, and the position of the movement destination of the hand of theuser US.

The user US can rotate the spherical image VSS1 in any direction to anyposition at any speed by moving the hand in any direction to anyposition at any speed. That is, the user US can rotate the sphericalimage VSS1 by the movement of the hand. The image processor 111 movesthe shooting image IM1 corresponding to the rotation of the sphericalimage VSS1.

The image processor 111 can determine to which position on thecoordinate of the spherical image VSS1 the zenith ZE of the user USbefore the user US rotates the spherical image VSS1 has moved by theuser US rotating the spherical image VSS1. An image processor 111calculates a change amount of the spherical image VSS1 before and afterthe user US rotates the spherical image VSS1 based on the movingdirection of the zenith ZE and the position of the moving destination onthe coordinate of the spherical image VSS1.

The amount of change of the spherical image VSS1 corresponds to theamount of rotation (rotation angle) of the spherical image VSS1 obtainedby synthesizing the amount of rotation (rotation angle) about the Xaxis, the amount of rotation (rotation angle) about the Y axis, and theamount of rotation (rotation angle) about the Z axis in the sphericalimage VSS1. The image processor 111 stores the variation of thespherical image VSS1 as a correction value CV1. That is, the correctionvalue CV1 is calculated based on the rotation direction of the sphericalimage VSS1 and the moving amount or moving angle of the zenith ZE(rotation angle of the spherical image VSS1).

The image processor 111 may store the coordinates on the spherical imageVSS1 of the zenith ZE as the correction value CV1 after the user USrotates the spherical image VSS1. The image processor 111 may store thecorrection value CV1 in a built-in memory or an external memory, or mayoutput the correction value CV1 to the server 106 via the secondcommunicator 113 and the network. The server 106 stores the correctionvalue CV1 in a built-in memory or an external memory in association withthe image display 104 or the user US.

The image processor 111 detects the horizontal direction of the shootingimage IM1. However, it is difficult for the image processor 111 todetect the horizontal direction depending on the shooting image IM1, andthe horizontal direction may be erroneously detected. In addition, thehorizontal or zenith designation may be shifted due to problems such assystem synchronization. When the horizontal direction of the shootingimage IM1 is erroneously detected or when the horizontal or zenithdesignation is deviated, the user US feels a sense of incongruitybecause the shooting image IM1 displayed on the image display 104 doesnot coincide with the gravity sensation of the user US.

An example of an image adjustment method according to the firstembodiment will be described with reference to flowcharts illustrated inFIGS. 3 and 4. Specifically, an example of a method for adjusting thehorizontal position of the shooting image IM1 and the composite imageCM1 will be described. The image display 104 is mounted on the head ofthe user US, and the controller 105 is attached on the hand of the userUS. The image display 104 displays the shooting image IM1 and thecomposite image CM1. At this point, the composite image CM1 may not bedisplayed.

If the user US determines that the shooting image IM1 displayed on theimage display 104 is not horizontal, in FIG. 3, the user US operates thecontroller 105 so that the image processor 111 causes the image display104 to display a setting screen in step S101.

When the user US operates the controller 105 to select a predetermineditem (e.g. horizontal adjustment items) displayed on the setting screen,the image processor 111 shifts the processing to a predeterminedprocessing mode corresponding to the selected item in step S102. Whenthe item of horizontal adjustment is selected, the image processor 111shifts the processing to a processing mode (horizontal adjustment mode)for adjusting the horizontal of the shooting image IM1.

In step S103, the image processor 111 acquires the spherical image VSS1generated by the image generator 112, and displays the spherical imageVSS1 on the image display 104. In the image display 104, the shootingimage IM1, the spherical image VSS1, and the composite image CM1 aremixed and displayed. When the user US rotates the spherical image VSS1so that the shooting image IM1 becomes horizontal, the image processor111 moves the shooting image IM1 displayed on the image display 104 instep S104 in accordance with the rotation of the spherical image VSS1.The user US may rotate the spherical image VSS1 a plurality of timesuntil the shooting image IM1 becomes horizontal.

When the user US determines that the shooting image IM1 is horizontal,the controller 105 is operated so that the image processor 111terminates the display of the spherical image VSS1 in step 5105 andcauses the image display 104 to display a setting screen. By making theshooting image IMI displayed on the image display 104 horizontal, thezenith of the camera 102 and the zenith of the user US can be matched.

When the user US operates the controller 105 to select a predetermineditem (e.g. end item) displayed on the setting screen, the imageprocessor 111 shifts the processing to a predetermined processing modecorresponding to the selected item in step S106. When the item to beended is selected, the image processor 111 shifts the processing to aprocessing mode (exit mode) for ending the horizontal adjustment.

In step S107, the image processor 111 acquires the amount of rotation(rotation angle) before and after the rotation of the spherical imageVSS1 as the amount of change of the spherical image VSS1. In step S108,the image processor 111 stores the amount of change in the sphericalimage VSS1 as the correction value CV1, and ends the process.

If the composite image CM1 is not displayed in steps S101 to S108 ofFIG. 3, by the user US operating the controller 105, the image adjustor110 (more specifically, the image processor 111) can acquire thecomposite image CM1 generated by the image generator 112, synthesize thecomposite image CM1 with the shooting image IM1 displayed on the imagedisplay 104, and display the synthesized image.

FIG. 5 schematically illustrates the relationship between the horizontalplane PHP1 of the shooting image IM1 horizontally adjusted by the userUS and the horizontal plane SHP1 of the composite image CM 1 acquired bythe image processor 111 from the image generator 112. When thehorizontal plane PHP1 of the shooting image IM1 deviates from thehorizontal plane SHP1 of the composite image CM1, the user US feels asense of incongruity.

In FIG. 4, the image processor 111 determines in step S111 whether ornot the shooting image IM1 has been horizontally adjusted. The imageprocessor 111 can determine whether or not the shooting image IM1 hasbeen horizontally adjusted based on the history of the rotation of thespherical image VSS1, the correction value CV1, or the like.

If it is determined in step S111 that the shooting image IM has beenhorizontally adjusted (YES), the image processor 111 acquires the storedcorrection value CV1 in step S112. In step S113, the image processor 111rotates the coordinate axis of the composite image CM1 based on thecorrection value CV1, and makes the horizontal plane SHP1 of thecomposite image CM1 coincide with the horizontal plane PHP1 of theshooting image IM1. In step S114, the image processor 111 performsarithmetic processing on the composite image CM1 based on the correctedhorizontal plane SHP1 to adjust the image.

In step S115, the image processor 111 synthesizes the adjusted compositeimage CM1 with the shooting image IM1 and displays the synthesized imageon the image display 104. If it is determined in step S111 that theshooting image IM1 is not horizontally adjusted (NO), the imageprocessor 111 shifts the processing to step S115.

In the image adjustment system 101, the image adjustor 110, and theimage adjustment method according to the first embodiment, the imagedisplay 104 displays the spherical image VSS1. With the image adjustmentsystem 101, the image adjustor 110, and the image adjustment methodaccording to the first embodiment, when the horizontal direction of theshooting image IM1 is erroneously detected or when the horizontaldirection or the zenith ZE designation is deviated, the user US operatesthe controller 105 to rotate the spherical image VSS1, so that theshooting image IM1 displayed on the image display 104 can be adjusted tobe horizontal.

Therefore, with the image adjustment system 101, the image adjustor 110,and the image adjustment method according to the first embodiment, whenthe horizontal direction of the shooting image IM1 is erroneouslydetected or the horizontal or zenith designation is shifted, the user UScan easily correct the horizontal or zenith of the shooting image IM1.

With the image adjustment system 101, the image adjustor 110, and theimage adjustment method according to the first embodiment, when thecorrection value CV1 is stored, the image processor 111 reads thecorrection value CV1, adjusts the shooting image IM1 imaged by thecamera 102 based on the correction value CV1, and displays the image onthe image display 104.

In the image adjustment system 101, the image adjustor 110, and theimage adjustment method according to the first embodiment, when theshooting image IM1 is horizontally adjusted, the coordinate axis of thecomposite image CM1 is rotated based on the correction value CV1, andthe horizontal plane SHP1 of the composite image CM1 is made to coincidewith the horizontal plane PHP1 of the shooting image IM1. With the imageadjustment system 101, the image adjustor 110, and the image adjustmentmethod according to the first embodiment, the shooting image IM1 and thecomposite image CM1 can be displayed on the image display 104 withoutcausing a sense of incongruity to the user US.

Second Embodiment

With reference to FIG. 1, a configuration example of an image adjustmentsystem 201 according to a second embodiment will be described. The imageadjustment system 201 according to the second embodiment includes acamera 202, a first communicator 203, an image display 204, a controller205, an image adjustor 210, and a server 206. The image adjustor 210includes an image processor 211, an image generator 212, and a secondcommunicator 213.

The camera 202, the first communicator 203, the image display 204, thecontroller 205, the image adjustor 210, and the server 206 of the secondembodiment correspond to the camera 102, the first communicator 103, theimage display 104, the controller 105, the image adjustor 110, and theserver 106 of the first embodiment. The image processor 211, the imagegenerator 212, and the second communicator 213 of the second embodimentcorrespond to the image processor 111, the image generator 112, and thesecond communicator 113 of the first embodiment.

The image adjustor 210 can acquire a shooting image IM2 imaged by thecamera 202 through the first communicator 203, the second communicator213, and a network. The server 206 may acquire a shooting image IM2imaged by the camera 202 via the first communicator 203 and the network,and the image adjustor 210 may acquire the shooting image IM2 from theserver 206 via the network and the second communicator 213.

The shooting image IM2 acquired by the image adjustor 210 is input tothe image processor 211. The image processor 211 analyzes the shootingimage IM2 to recognize the horizontal direction of the shooting imageIM2. The image processor 211 may analyze the shooting image IM2 torecognize the vertical direction in the shooting image IM2, or mayrecognize the horizontal direction and the vertical direction. The imageadjustor 210 corrects distortion of the shooting image IM2, executesimage processing such as adjustment of the horizontal position of theshooting image IM2, and outputs the image-processed shooting image IM2to the image display 204.

The server 206 may correct distortion of the shooting image IM2 imagedby the camera 202, perform image processing such as adjusting thehorizontal position of the shooting image IM2, and output theimage-processed shooting image IM2 to the image adjustor 210. The imagedisplay 204 displays the shooting image IM2 subjected to imageprocessing by the image adjustor 210 or the server 206. By attaching theimage display 204 to the head of the user US, the user US can watch theshooting image IM2 subjected to image processing by the image adjustor210 or the server 206.

The image display 204 generates attitude information PN2 on the basis ofa direction in which the user US faces and a state such as the attitudeof the user US in a state of being mounted on the head of the user US.The image processor 211 acquires the attitude information PN2 from animage display 204. That is, the image processor 211 acquires theattitude information PN2 based on the attitude of the image display 204.The image processor 211 displays, on the basis of the attitudeinformation PN2, an image of an area corresponding to the direction inwhich the user US faces and the attitude of the user US from theshooting image IM2 imaged by the camera 202 on the image display 204.

The controller 205 generates instruction information NN2 on the basis ofthe state such as the movement or attitude of the hand of the user US inthe state of being attached on the hand of the user US. The imageprocessor 211 acquires the instruction information NN2 from thecontroller 205. Based on the instruction information NN2, the imageprocessor 211 can change or adjust the shooting image IM2 displayed onthe image display 204.

The image generator 212 generates a composite image CM2 being CG to besynthesized with the shooting image IM2. The composite image CM2 is animage of a character such as an avatar. The image generator 212 storesthe composite image CM2 in a built-in memory or an external memory. Theimage processor 211 may acquire the composite image CM2 generated by theimage generator 212 and output it to the server 206 via the secondcommunicator 213 and the network. The server 206 stores the compositeimage CM2 in a built-in memory or an external memory in association withthe image display 204 or the user US.

The image processor 211 acquires the composite image CM2 from the imagegenerator 212 or the server 206 based on the instruction informationNN2, and synthesizes the composite image CM2 with the shooting image IM2displayed on the image display 204. Therefore, the image display 204 candisplay the shooting image IM2 imaged by the camera 202 and thecomposite image CM2 acquired from the image generator 212 or the server206 and synthesized with the shooting image IM2.

The image generator 212 generates a spherical image VSS2 which is avirtual image constituted of a spherical surface which is CG. The imagegenerator 212 stores the spherical image VSS2 in a built-in memory or anexternal memory. On the basis of the instruction information NN2, theimage processor 211 acquires the spherical image VSS2 generated by theimage generator 212 and displays the spherical image VSS2 on the imagedisplay 204.

The spherical image VSS2 of the second embodiment corresponds to thespherical image VSS1 of the first embodiment. By moving the hand onwhich the controller 205 is attached to a position corresponding to thespherical image VSS2 displayed on the image display 204, the user USfeels as if the hand of the user US is in contact with the sphericalimage VSS2.

The controller 205 may have an actuator placed in contact with the handof the user US. On the basis of the instruction information NN2, theimage processor 211 operates the actuator when it is determined that thehand of the user US has moved to a position corresponding to thespherical image VSS2. When the actuator applies pressure to the hand ofthe user US, the user US can actually feel the feeling that the hand isin contact with the spherical image VSS2.

In a state where the spherical image VSS2 is displayed on the imagedisplay 204, when the user US moves the hand on which the controller 205is attached in an arbitrary direction, the image processor 211 performsimage processing based on the instruction information NN2 so that thespherical image VSS2 and the shooting image IM2 displayed on the imagedisplay 204 move in accordance with the moving direction of the hand,the moving speed, and the position of the movement destination of theuser US.

The user US can rotate the spherical image VSS2 in any direction to anyposition at any speed by moving the hand in any direction to anyposition at any speed. That is, the user US can rotate the sphericalimage VSS2 by the movement of the hand. The image processor 211 movesthe shooting image IM2 corresponding to the rotation of the sphericalimage VSS2.

The image processor 211 can determine to which position on thecoordinate of the spherical image VSS2 the zenith ZE of the user USbefore the user US rotates the spherical image VSS2 has moved by theuser US rotating the spherical image VSS2. The image processor 211calculates a change amount of the spherical image VSS2 before and afterthe user US rotates the spherical image VSS2 based on the movingdirection of the zenith ZE and the position of the moving destination onthe coordinate of the spherical image VSS2. The variation of thespherical image VSS2 of the second embodiment corresponds to thevariation of the spherical image VSS1 of the first embodiment.

The image processor 211 stores the amount of change in the sphericalimage VSS2 as a correction value CV2. The correction value CV2 of thesecond embodiment corresponds to the correction value CV1 of the firstembodiment. The image processor 211 may store the coordinates on thespherical image VSS2 of the zenith ZE after the user US rotates thespherical image VSS2 as the correction value CV2. The image processor211 may store the correction value CV2 in a built-in memory or anexternal memory, or may output the correction value CV2 to the server206 via the second communicator 213 and the network. The server 206stores the correction value CV2 in a built-in memory or an externalmemory in association with the image display 204 or the user US.

The image processor 211 detects the horizontal direction of the shootingimage IM2. However, it is difficult for the image processor 211 todetect the horizontal direction depending on the shooting image IM2, andthe horizontal direction may be erroneously detected. In addition, thehorizontal or zenith designation may be shifted due to problems such assystem synchronization. When the horizontal direction of the shootingimage IM2 is erroneously detected or when the horizontal or zenithdesignation is deviated, the user US feels a sense of incongruitybecause the shooting image IM2 displayed on the image display 204 doesnot coincide with the gravity sensation of the user US.

An example of an image adjustment method according to the secondembodiment will be described with reference to flowcharts illustrated inFIGS. 6A, 6B, and 7. Specifically, an example of a method for adjustingthe horizontal position of the shooting image IM2 and the compositeimage CM2 will be described. The image display 204 is mounted on thehead of the user US, and the controller 205 is mounted on the hand ofthe user US. The image display 204 displays the shooting image IM2 andthe composite image CM2. The composite image CM2 may not be displayed.

If the user US determines that the shooting image IM2 displayed on theimage display 204 is not horizontal, in FIG. 6A, the user US operatesthe controller 205 so that the image processor 211 causes the imagedisplay 204 to display a setting screen in step S201.

When the user US operates the controller 205 to select a predetermineditem (e.g. horizontal adjustment item) displayed on the setting screen,the image processor 211 shifts the processing to a predeterminedprocessing mode corresponding to the selected item in step S202. Whenthe item of horizontal adjustment is selected, the image processor 211shifts the processing to a processing mode (horizontal adjustment mode)for adjusting the horizontal of the shooting image IM2.

In step S203, the image processor 211 acquires the spherical image VSS2generated by the image generator 212, and displays the spherical imageVSS2 on the image display 204. In the image display 204, the shootingimage IM2 and the spherical image VSS2 are mixed and displayed. In stepS204, the image processor 211 determines whether or not the compositeimage CM2 is synthesized with the shooting image IM2 and displayed onthe image display 204.

When it is determined in step S204 that the composite image CM2 issynthesized with the shooting image IM2 and displayed on the imagedisplay 204 (YES), the image processor 211 executes, in step S205, aprocess for separating the horizontal plane SHP2 of the composite imageCM2 from the horizontal plane PHP2 of the shooting image IM2. FIG. 8schematically illustrates a state in which the horizontal plane SHP2 ofthe composite image CM2 and the horizontal plane PHP2 of the shootingimage IM2 are separated.

If it is determined in step S204 that the composite image CM2 issynthesized with the shooting image IM2 and is not displayed on theimage display 204 (NO), the image processor 211 advances the process tostep S206.

The user US rotates the spherical image VSS2 so that the shooting imageIM2 is horizontal. Since the horizontal plane SHP2 of the compositeimage CM2 is separated from the horizontal plane PHP2 of the shootingimage IM2, the composite image CM2 is not rotated even if the sphericalimage VSS2 is rotated. Therefore, the user US can easily correct thehorizontal position of the shooting image IM2 without being confused bythe rotation of the composite image CM2.

In step S205, the image processor 211 may hide the composite image CM2.Since the composite image CM2 is not obstructed by making the compositeimage CM2 invisible, the user US can easily correct the horizontalposition of the shooting image IM2.

In step S206, the image processor 211 moves the shooting image IM2displayed on the image display 204 in accordance with the rotation ofthe spherical image VSS2. The user US may rotate the spherical imageVSS2 a plurality of times until the shooting image IM2 becomeshorizontal.

In FIG. 6B, when the user US determines that the shooting image IM2 ishorizontal, the controller 205 is operated so that the image processor211 terminates the display of the spherical image VSS2 in step S207 andcauses the image display 204 to display the setting screen. By makingthe shooting image IM2 displayed on the image display 204 horizontal,the zenith of the camera 202 and the zenith of the user US can bematched.

When the user US operates the controller 205 to select a predetermineditem (e.g. end item) displayed on the setting screen, the imageprocessor 211 shifts the processing to a predetermined processing modecorresponding to the selected item in step S208. When the item to beended is selected, the image processor 211 shifts the processing to aprocessing mode (end horizontal adjustment mode) for ending thehorizontal adjustment.

In step S209, the image processor 211 acquires the amount of rotationbefore and after the rotation (rotation angle) of the spherical imageVSS2 as the amount of change of the spherical image VSS2. In step S210,the image processor 211 stores the change amount of the spherical imageVSS2 as the correction value CV2.

If the composite image CM2 is not displayed in steps S201 to S210 ofFIGS. 6A and 6B, the user US operates the controller 205 so that theimage adjustor 210 (more specifically, the image processor 211) acquiresthe composite image CM2 generated by the image generator 212 anddisplays it on the image display 204.

An image adjustor 210 (more specifically, the image processor 211)analyzes a shooting image IM2 imaged by a camera 202 to recognize ahorizontal direction in the shooting image IM2. After the user USrotates the spherical image VSS2 to adjust the horizontal position ofthe shooting image IM2, when the image adjustor 210 acquires thecomposite image CM2 generated by the image generator 212 and displaysthe composite image CM2 on the image display 204, the horizontal planeSHP2 of the composite image CM2 does not coincide with the horizontalplane PHP2 of the shooting image IM2. When the horizontal plane PHP2 ofthe shooting image IM2 deviates from the horizontal plane SHP2 of thecomposite image CM2, the user US feels a sense of incongruity.

In FIG. 7, the image processor 211 determines in step S211 whether ornot the composite image CM2 is synthesized with the shooting image IM2and displayed on the image display 204. If it is determined that thecomposite image CM2 is synthesized with the shooting image IM2 anddisplayed on the image display 204 (YES), the image processor 211acquires the stored correction value CV2 in step S212.

In step S213, the image processor 211 rotates the coordinate axis of thecomposite image CM2 based on the correction value CV2, and makes thehorizontal plane SHP2 of the composite image CM2 coincide with thehorizontal plane PHP2 of the shooting image IM2 horizontally adjusted bythe user US. In step S214, the image processor 211 performs arithmeticprocessing on the composite image CM2 based on the corrected horizontalplane SHP2 to adjust the image. FIG. 8 schematically illustrates a statein which the horizontal plane SHP2 of the composite image CM2 coincideswith the horizontal plane PHP2 of the shooting image IM2.

If it is determined in step S211 that the shooting image IM2 is nothorizontally adjusted by the user US (NO), the image processor 211shifts the processing to step S215. In step S215, the image processor211 synthesizes the adjusted composite image CM2 with the shooting imageIM2 and displays the composite image CM2 on the image display 204. Theuser US can watch the shooting image IM2 and the composite image CM2which are displayed on the image display 204 and whose horizontal linescoincide with each other.

In the image adjustment system 201, the image adjustor 210, and theimage adjustment method according to the second embodiment, the imagedisplay 204 displays the spherical image VSS2. With the image adjustmentsystem 201, the image adjustor 210, and the image adjustment methodaccording to the second embodiment, when the horizontal direction of theshooting image IM2 is erroneously detected or when the horizontaldirection or the zenith ZE designation is deviated, the user US operatesthe controller 205 to rotate the spherical image VSS2, so that theshooting image IM2 displayed on the image display 204 can be adjusted tobe horizontal.

Therefore, with the image adjustment system 201, the image adjustor 210,and the image adjustment method according to the second embodiment, whenthe horizontal direction of the shooting image IM2 is erroneouslydetected or the horizontal or zenith designation is shifted, the user UScan easily correct the horizontal or zenith of the shooting image IM2.

With the image adjustment system 201, the image adjustor 210, and theimage adjustment method according to the second embodiment, when thecorrection value CV2 is stored, the image processor 211 reads thecorrection value CV2, adjusts the shooting image IM2 imaged by thecamera 202 based on the correction value CV2, and displays the image onthe image display 204.

In the image adjustment system 201, the image adjustor 210, and theimage adjustment method according to the second embodiment, thehorizontal plane SHP2 of the composite image CM2 and the horizontalplane PHP2 of the shooting image IM2 are separated, the horizontal planeof the shooting image IM2 is adjusted, and the horizontal plane of thecomposite image CM2 is adjusted based on the adjustment result(correction value CV2). Since the horizontal plane SHP2 of the compositeimage CM2 is separated from the horizontal plane PHP2 of the shootingimage IM2, the composite image CM2 is not rotated even when the user USrotates the spherical image VSS2. Therefore, the user US can easilycorrect the horizontal position of the shooting image IM2.

With the image adjustment system 201, the image adjustor 210, and theimage adjustment method according to the second embodiment, when theshooting image IM2 is horizontally adjusted, the coordinate axis of thecomposite image CM2 is rotated based on the correction value CV2, andthe horizontal plane SHP2 of the composite image CM2 is made to coincidewith the horizontal plane PHP2 of the shooting image IM2. Therefore, theuser US can watch the shooting image IM2 and the composite image CM2displayed on the image display 204 without feeling a sense ofincongruity.

Third Embodiment

With reference to FIG. 9, a configuration example of a image adjustmentsystem 301 according to a third embodiment will be described. The imageadjustment system 301 according to the third embodiment includes acamera 302, a first communicator 303, a plurality of image displays 304,a plurality of controllers 305, a plurality of image adjustors 310, anda server 306. Each of the image adjustors 310 includes an imageprocessor 311, an image generator 312, and a second communicator 313.

The camera 302, the first communicator 303, each of the image displays304, each of the controllers 305, each of the image adjustors 310, andthe server 306 of the third embodiment correspond to the camera 102, thefirst communicator 103, the image display 104, the controller 105, theimage adjustor 110, and the server 106 of the first embodiment,respectively. The image processor 311, the image generator 312, and thesecond communicator 313 of the third embodiment correspond to the imageprocessor 111, the image generator 112, and the second communicator 113of the first embodiment, respectively.

FIG. 9 illustrates two image displays 304, two controllers 305, and twoimage adjustors 310, respectively, for the sake of explanation. Theimage displays 304, the controllers 305, and the image adjustor 310 maybe three or more. The image display 304 used by a first user USa isreferred to as a first image display 304 a, the controller 305 isreferred to as a first controller 305 a, and the image adjustor 310 isreferred to as a first image adjustor 310 a. The image display 304 usedby a second user USb is referred to as a second image display 304 b, thecontroller 305 is referred to as a second controller 305 b, and theimage adjustor 310 is referred to as a second image adjustor 310 b.

In the first image adjustor 310 a, the image processor 311 is referredto as an image processor 311 a, the image generator 312 is referred toas an image generator 312 a, and the second communicator 313 is referredto as a second communicator 313 a. In the second image adjustor 310 a,the image processor 311 is referred to as an image processor 311 a, theimage generator 312 is referred to as an image generator 312 a, and thesecond communicator 313 is referred to as a second communicator 313 a.

The first image adjustor 310 a can acquire a shooting image IM3 imagedby the camera 302 through the first communicator 303, the secondcommunicator 310 a of the first image adjustor 313 a and the network.The second image adjustor 310 b can acquire the shooting image IM3imaged by the camera 302 through the first communicator 303, the secondcommunicator 310 b of the second image adjustor 313 b and the network.

The server 306 may acquire the shooting image IM3 shooting by the camera302 via the first communicator 303 and the network, the first imageadjustor 310 a may acquire the shooting image IM3 from the server 306via the network and the second communicator 310 a of the first imageadjustor 313 a, and the second image adjustor 310 b may acquire theshooting image IM3 from the server 306 via the network and the secondcommunicator 310 b of the second image adjustor 313 b.

The server 306 can identify the first user USa or the first imagedisplay 304 a by performing a login operation by the first user USa orby connecting the first image display 310 a to the network through thesecond communicator 313 a of the first image adjustor 304 a. The server306 can specify the second user USb or the second image display 304 b byexecuting a login operation by the second user USb or by connecting thesecond image display 310 b to the network through the secondcommunicator 313 b of the second image adjustor 304 b.

The processes performed by the first image adjustor 310 a, the firstimage display 304 a, and the first controller 305 a will be describedbelow. The symbols in parentheses indicate the processes performed bythe second image adjustor 310 b, the second image display 304 b, and thesecond controller 305 b.

The shooting image IM3 acquired by the image adjustor 310 a (310 b) isinput to the image processor 311 a (311 b). The image processor 311 a(311 b) analyzes the shooting image IM3 to recognize the horizontaldirection of the shooting image IM3. The image processor 311 a (311 b)may analyze the shooting image IM3 to recognize the vertical directionin the shooting image IM3, or may recognize the horizontal direction andthe vertical direction.

The image adjustor 310 a (310 b) corrects the distortion of the shootingimage IM3, performs image processing such as adjusting the horizontalposition of the shooting image IM3, and outputs the image-processedshooting image IM3 to the image display 304 a (304 b).

The server 306 may correct distortion of the shooting image IM3 imagedby the camera 302, perform image processing such as adjusting thehorizontal position of the shooting image IM3, and output theimage-processed shooting image IM3 to the image adjustor 310 a (310 b).The image display 304 a (304 b) displays the shooting image IM3subjected to image processing by the image adjustor 310 a (310 b) or theserver 306.

When the user USa (USb) mounts the image display 304 a (304 b) on thehead thereof, the user can watch the shooting image IM3 subjected toimage processing by the image adjustor 310 a (310 b) or the server 306.

While the image display 304 a (304 b) is mounted on the head of the userUSa (USb), the image display 304 a (304 b) generates attitudeinformation PN3 a (PN3 b) based on the direction in which the user USa(USb) faces and the attitude or the like of the user USa (USb). Theimage processor 311 a (311 b) acquires the attitude information PN3 a(PN3 b) from the image display 304 a (304 b). That is, the imageprocessor 311 a (311 b) acquires the attitude information PN3 a (PN3 b)based on the attitude of the image display 304 a (04 b).

The image processor 311 a (311 b), based on the attitude information PN3a (PN3 b), displays, from the shooting image IM3 imaged by the camera302, an image of an area corresponding to the direction in which theuser USa (USb) faces and the state such as the attitude of the user USa(USb) on the image display 304 a (304 b).

While the controller 305 a (305 b) is attached to the hand of the userUSa (USb), the controller 305 a (305 b) generates instructioninformation NN3 a (NH3 b) based on a state such as the movement orattitude of the hand of the user USa (USb). The image processor 311 a(311 b) acquires the instruction information NN3 a HN3 b) from thecontroller 305 a (305 b). The image processor 311 a (311 b) can changeor adjust the shooting image IM3 displayed on the image display 304 a(304 b) based on the instruction information NN3 a (NN3 b).

The image generator 312 a (312 b) generates a composite image CM3 b(CM3a) which is a CG to be synthesized with the shooting image IM3. Thecomposite image CM3 a (CM3 b) is an image of a character such as anavatar. The composite image CM3 a generated by the second imagegenerator 312 b is the avatar of the first user USa, and the compositeimage CM3 b generated by the first image generator 312 a is the avatarof the second user USb. When the user USa (USb) operates the controller305 a (305 b), the image generator 312 a (312 b) generates a compositeimage CM3 b (CM3 a), which is the avatar of the user USb (USa), andstores the composite image CM3 b in a built-in memory or an externalmemory.

The image processor 311 a (311 b) may acquire the composite image CM3 b(CM3a) generated by the image generator 312 a (312 b) and output it tothe server 306 via the second communicator 313 a (313 b) and thenetwork.

When the user USa (USb) operates the controller 305 a (305 b), the imagegenerator 312 a (312 b) may generate a composite image CM3 a (CM3b)which is the avatar of the user USa (USb). The image processor 311 a(311 b) may acquire the composite image CM3 a (CM3 b) generated by theimage generator 312 a (312 b) and output it to the server 306 via thesecond communicator 313 a (313 b) and the network. That is, the firstuser USa may set the avatar of the second user USb or may set its ownavatar. The second user USb may set the avatar of the first user USa ormay set its own avatar.

The server 306 stores the composite image CM3 a (CM3 b) in a built-inmemory or an external memory in association with the user USa (USb). Theserver 306 may automatically set the composite image CM3 a (CM3 b) inassociation with the user USa (USb) and store it in the built-in memoryor the external memory.

Based on the instruction information NN3 a (NN3 b), the image processor311 a (311 b) acquires the composite image CM3 b (CM3 a) from the imagegenerator 312 a (312 b) or the server 306, and synthesizes the compositeimage CM3 b (CM3 a) with the shooting image IM3 displayed on the imagedisplay 304 a (304 b). Therefore, the image display 304 a (304 b) candisplay the shooting image IM3 imaged by the camera 302 and thecomposite image CM3 b (CM3a) acquired from the image generator 312 a(312 b) or the server 306 and synthesized with the shooting image IM3.

The image generator 312 a (312 b) generates a spherical image VSS3 a(VSS3 b) that is a virtual image formed of a spherical surface that isCG. The image generator 312 a (312 b) stores the spherical image VSS3 a(VSS3 b) in a built-in memory or an external memory. The image processor311 a (311 b) acquires, based on the instruction information NN3 a (NN3b), the spherical image VSS3 a (VSS3 b) generated by the image generator312 a (312 b) and displays it on the image display 304 a (304 b).

The spherical image VSS3 a (VSS3 b) of the third embodiment correspondsto the spherical image VSS1 of the first embodiment. The user USa (USb)moves the hand on which the controller 305 a (305 b) is mounted to aposition corresponding to the spherical image VSS3 a (VSS3 b) displayedon the image display 304 a (304 b), and feels as if the hand of the userUSa (USb) is in contact with the spherical image VSS3 a (VSS3 b).

The controller 305 a (305 b) may have an actuator disposed in a portionof the user USa (USb) in contact with the hand. Based on the instructioninformation NN3 a (NN3 b), the image processor 311 a (311 b) operatesthe actuator when it is determined that the hand of the user USa (USb)has moved to a position corresponding to the spherical image VSS3 a(VSS3 b). When the actuator applies pressure to the hand of the user USa(USb), the user USa (USb) can feel the sensation that the hand contactsthe spherical image VSS3 a (VSS3 b).

While the spherical image VSS3 a (VSS3 b) is displayed on the imagedisplay 304 a (304 b), when the user USa (USb) moves the hand on whichthe controller 305 a (305 b) is attached in an arbitrary direction, theimage processor 311 a (311 b), based on the instruction information NN3a (NN3 b), performs image processing so that the spherical image VSS3 a(VSS3b) and the shooting image IM3 displayed on the image display 304 a(304 b) move in accordance with the moving direction, moving speed andposition of the hand of the user USa (USb).

The user USa (USb) can rotate the spherical image VSS3 a (VSS3 b) in anarbitrary direction to an arbitrary position at an arbitrary speed bymoving the hand in an arbitrary direction to an arbitrary position at anarbitrary speed. That is, the user USa (USb) can rotate the sphericalimage VSS3 a (VSS3 b) by the hand movement. The image processor 311 a(311 b) moves the shooting image IM3 in accordance with the rotation ofthe spherical image VSS3 a (VSS3 b).

The image processor 311 a (311 b) can determine to which position on thecoordinate of the spherical image VSS3 a (VSS3 b) the zenith ZE of theuser USa before the user USa rotates the spherical image VSS3 a (VSS3 b)has moved by the user USa (USb) rotating the spherical image VSS3 a(VSS3 b).

The image processor 311 a (311 b) calculates, based on the movingdirection of the zenith ZE and the position of the moving destination onthe coordinate of the spherical image VSS3 a (VSS3 b), the amount ofchange in the spherical image VSS3 a (VSS3 b) between the time beforeand after the user USa (USb) rotates the spherical image VSS3 a (VSS3b). The change amount of the spherical image VSS3 a (VSS3 b) of thethird embodiment corresponds to the change amount of the spherical imageVSS1 of the first embodiment.

The image processor 311 a (311 b) stores the amount of change in thespherical image VSS3 a (VSS3 b) as the correction value CV3 a (CV3 b).The correction value CV3 a (CV3 b) of the third embodiment correspondsto the correction value CV1 of the first embodiment. The image processor311 a (311 b) may store the coordinates on the spherical image VSS3 a(VSS3 b) of the zenith ZE after the user USa (USb) rotates the sphericalimage VSS3 a (VSS3 b) as the correction value CV3 a (CV3 b).

The image processor 311 a (311 b) may store the correction value CV3 a(CV3 b) in a built-in memory or an external memory, or may output thecorrection value CV3 a (CV3 b) to the server 306 via the secondcommunicator 313 a (313 b) and the network. The server 306 stores thecorrection value CV3 a (CV3 b) in a built-in memory or an externalmemory in association with the image display 304 a (304 b) or the userUSa (USb).

The image adjustment system 301 can display the composite image CM3 b(CM3 a), which is an avatar of another user USb (USa), on the imagedisplay 304 a (304 b) mounted on the user USa (USb), by synthesizing thecomposite image CM3 b (CM3 a) with the shooting image IM3.

When the user USa (USb) watches the shooting image IM3 imaged by thecamera 302, the user USa (USb) feels as if the user USa (USb) iswatching around the camera 302 from the position thereof. In such acase, the image display 304 a (304 b) watched by the user USa (USb)displays the composite image CM3 b (CM3 a), which is an avatar ofanother user USb (USa), at the position of the user USa (USb), so thatthe user USa (USb) feels a sense of incongruity. When there are aplurality of other users US, avatars of the plurality of other usersUSb, USc, . . . are overlapped and displayed at the position of the userUSa.

In the server 306, positional relationships among the plurality of usersUSa, USb, . . . are preset. In other words, the server 306 storespositional setting information indicating positional relationships amonga plurality of users USa, USb, . . . The positional relationship amongthe plurality of users USa, USb, . . . may be automatically set by theserver 306, or may be independently set by the plurality of users USa,USb, . . .

FIG. 10 illustrates an example of the positional relationship betweenthe first user USa and the composite image CM3 b which is the avatar ofthe second user USb when the first user USa is watching the first imagedisplay 304 a. FIG. 11 illustrates an example of the positionalrelationship between the second user USb and the composite image CM3 awhich is the avatar of the first user USa when the second user USb iswatching the second image display 304 b.

The black dots illustrated in FIGS. 10 and 11 indicate the position ofthe camera 302. Arrows illustrated in FIGS. 10 and 11 indicate themoving direction of the camera 302. That is, FIG. 10 illustrates a statein which the first user USa faces the moving direction of the camera302, and FIG. 11 illustrates a state in which the second user USb facesthe moving direction of the camera 302. Therefore, in each of FIGS. 10and 11, the first user USa and the second user USb are watching the sameshooting image IM3.

In such a case, when the composite image CM3 b is displayed at aposition away from the first user USa or the composite image CM3 a isdisplayed at a position away from the second user USb, both the firstuser USa and the second user USb feel uncomfortable. Therefore, it isdesirable to display the composite image CM3 b (CM3 a) at a positionwithin a predetermined range RG from the position of the user USa (USb)(Camera 302).

As illustrated in FIG. 10, in the server 306, for example, in the firstuser USa or the first image display 304 a, the positional relationshipbetween the first user USa and the second user USb is set so that thecomposite image CM3 b is arranged side by side on the right side of thefirst user USa. As illustrated in FIG. 11, in the server 306, forexample, in the second user USb or the second image display 304 b, thepositional relationship between the first user USa and the second userUSb is set so that the composite image CM3 a is arranged side by side onthe left side of the second user USb.

That is, in FIGS. 10 and 11, the positional relationship between thefirst user USa and the second user USb is set so that when the movingdirection of the camera 302 is the forward direction of the first userUSa and the second user USb, the second user USb is arranged side byside on the right side of the first user USa. Therefore, the first userUSa always watches the second user USb on the right side, and the seconduser USb always watches the first user USa on the left side. When thecamera 360, which is a camera 302, is stationary, there is no referencefor the front, back, right, and left, so that it is desirable todetermine the positional relationship between the first user USa and thesecond user USb at the time of initialization.

With reference to the flowcharts illustrated in FIGS. 12 and 13, anexample of an image adjustment method according to the third embodimentwill be described. Concretely, an example of a method for synthesizingthe composite image CM3 with the shooting image IM3 will be described.The processes performed by the first image adjustor 310 a, the firstimage display 304 a, and the first controller 305 a will be describedbelow. The symbols in parentheses indicate the processes performed bythe second image adjustor 310 b, the second image display 304 b, and thesecond controller 305 b.

The image display 304 a (304 b) is mounted on the head of the user USa(USb), and the controller 305 a (305 b) is attached on the hand of theuser USa (USb). The image display 304 a (304 b) displays the shootingimage IM3.

If the user USa (USb) determines that the shooting image IM3 displayedon the image display 304 a (304 b) is not horizontal, in FIG. 12, theuser USa (USb) operates the controller 305 a (305 b), whereby the imageprocessor 311 a (311 b) causes the image display 304 a (304 b) todisplay a setting screen in step S301.

When the user USa (USb) operates the controller 305 a (305 b) to selecta predetermined item (e.g. horizontal adjustment items) displayed on thesetting screen, the image processor 311 a (311 b) shifts the processingto a predetermined processing mode corresponding to the selected item instep S302. When the horizontal adjustment item is selected, the imageprocessor 311 a (311 b) shifts the processing to a processing mode(horizontal adjustment mode) for adjusting the horizontal of theshooting image IM3.

In step S303, the image processor 311 a (311 b) acquires the sphericalimage VSS3 a (VSS3 b) generated by the image generator 312 a (312 b) anddisplays it on the image display 304 a (304 b). The image display 304 a(304 b) displays a mixture of the shooting image IM3 and the sphericalimage VSS3 a (VSS3 b).

The user USa (USb) rotates the spherical image VSS3 a (VSS3 b) so thatthe shooting image IM3 is horizontal, and the image processor 311 a (311b) moves the shooting image IM3 displayed on the image display 304 a(304 b) in step S304 in accordance with the rotation of the sphericalimage VSS3 a (VSS3 b). The user USa (USb) may perform the operation ofrotating the spherical image VSS3 a (VSS3 b) a plurality of times untilthe shooting image IM3 becomes horizontal.

When the user USa (USb) determines that the shooting image IM3 ishorizontal, the controller 305 a (305 b) is operated so that the imageprocessor 311 a (311 b) terminates the display of the spherical imageVSS3 a (VSS3 b) in step S305 and causes the image display 304 a (304 b)to display the setting screen. By leveling the shooting image IM3displayed on the image display 304 a (304 b), the zenith of the camera302 and that of the user USa (USb) can be matched.

When the user USa (USb) operates the controller 305 a (305 b) to selecta predetermined item (For example, end item) displayed on the settingscreen, the image processor 311 a (311 b) shifts the processing to apredetermined processing mode corresponding to the selected item in stepS306. When the item to be ended is selected, the image processor 311 a(311 b) shifts the processing to a processing mode (exit mode) forending the horizontal adjustment.

In step S307, the image processor 311 a (311 b) acquires the amount ofrotation before and after the rotation (rotation angle) of the sphericalimage VSS3 a (VSS3 b) as the amount of change of the spherical imageVSS3 a (VSS3 b). In step S308, the image processor 311 a (311 b) storesthe amount of change in the spherical image VSS3 a (VSS3 b) as thecorrection value CV3 a (CV3 b), and ends the process.

With reference to a flowchart illustrated in FIG. 13, a case will bedescribed in which the composite image CM3 b, which is the avatar of thesecond user USb, is displayed on the first image display 304 a watchedby the first user USa in synthesizing with the shooting image IM3. Thereference numerals in parentheses indicate a case where the second imagedisplay 304 b watched by the second user USb displays the compositeimage CM3 a, which is the avatar of the first user USa, by synthesizingwith the shooting image IM3.

In FIG. 13, when the user USa (USb) operates the controller 305 a (305b), the image processor 311 a (311 b) causes the image display 304 a(304 b) to display the setting screen in step S311. The setting screendisplays an avatar selection item for selecting an avatar of anotheruser US, for example.

The user USa (USb) selects, for example, the user USb (USa) as the otheruser US. In step S312, the controller 305 a (305 b) generatesinstruction information NN3 a (USa) in accordance with the selection ofthe other user USb, and outputs the instruction information NN3 a (NN3b) to the image processor 311 a (311 b).

In step S313, the image processor 311 a (311 b) acquires the compositeimage CM3 b (CM3 a) based on the instruction information NN3 a (NN3 b).When the composite image CM3 b (CM3 a) is stored in the server 306, theimage processor 311 a (311 b) acquires the composite image CM3 b (CM3 a)from the server 306 via the second communicator 313 a (313 b) and thenetwork.

In step S314, the image processor 311 a (311 b) determines whether ornot the camera 302 is moving. If it is determined in step S314 that thecamera 302 is moving (YES), the image processor 311 a (311 b) recognizesthe moving direction of the camera 302 in step S315. The image processor311 a (311 b) determines, based on the shooting image IM3, whether ornot the camera 302 is moving, and can recognize the moving direction ofthe camera 302.

In step S316, the image processor 311 a (311 b) synthesizes thecomposite image CM3 b (CM3 a) with the shooting image IM3 so that thecomposite image CM3 b (CM3 a) is disposed, for example, on the right(left) side of the user USa (USb) with respect to the moving directionof the camera 302, and causes the image display 304 a (304 b) to displaythe composite image CM3 b (CM3 a).

When it is determined in step S314 that the camera 302 is not moving(NO), the image processor 311 a (311 b) determines that the camera 302is stationary, and in step S316, based on the positional relationshipbetween the first user USa and the second user USb determined in theinitial setting, the composite image CM3 b (CM3 a) is composited withthe shooting image IM3 and displayed on the image display 304 a (304 b).

In the image adjustment system 301, the image adjustor 310, and theimage adjustment method according to the third embodiment, the imagedisplay 304 displays the spherical image VSS3. With the image adjustmentsystem 301, the image adjustor 310, and the image adjustment methodaccording to the third embodiment, when the horizontal direction of theshooting image IM3 is erroneously detected or when the horizontaldirection or the designation of the zenith ZE is deviated, the user USoperates the controller 305 to rotate the spherical image VSS3, so thatthe shooting image IM3 displayed on the image display 304 can beadjusted to be horizontal.

Therefore, with the image adjustment system 301, the image adjustor 310,and the image adjustment method according to the third embodiment, whenthe horizontal direction of the shooting image IM3 is erroneouslydetected or the horizontal or zenith designation is shifted, the user UScan easily correct the horizontal or zenith of the shooting image IM3.

According to the image adjustment system 301, the image adjustor 310,and the image adjustment method according to the third embodiment, whenthe correction value CV3 is stored, the image processor 311 reads thecorrection value CV3, adjusts the shooting image IM3 imaged by thecamera 302 based on the correction value CV3, and displays the image onthe image display 304.

In the image adjustment system 301, the image adjustor 310, and theimage adjustment method according to the third embodiment, thepositional relationships of the plurality of users US are preset. As aresult, the shooting image IM3 displayed on the image display 304 a (304b) can display the composite image CM3 b (CM3 a), which is the avatar ofthe other user USb (USa), after being synthesized into a position basedon a preset positional relationship.

With the image adjustment system 301, the image adjustor 310, and theimage adjustment method according to the third embodiment, when the userUSa (USb) watches the shooting image IM3 imaged by the camera 302, thecomposite image CM3 b (CM3 a), which is the avatar of the other user USb(USa), is displayed in the vicinity of the user USa (USb), so that theuser USa (USb) can feel as if the user USa (USb) is watching the sameshooting image IM3 as the other user USb (USa) without feeling anydiscomfort.

Fourth Embodiment

With reference to FIG. 14, a configuration example of an imageadjustment system 401 according to a fourth embodiment will bedescribed. The image adjustment system 401 according to the fourthembodiment includes a plurality of cameras 402, a plurality of firstcommunicators 403, a plurality of image displays 404, a plurality ofcontrollers 405, a plurality of image adjustors 410, and a server 406.Each of the image adjustor 410 includes an image processor 411, an imagegenerator 412, and a second communicator 413.

Each of the cameras 402, each of the first communicators 403, and eachof the image displays 404 of the fourth embodiment correspond to thecamera 102, the first communicator 103, and the image display 104 of thefirst embodiment, respectively. Each of the controller 405, each of theimage adjustors 410, and the server 406 of the fourth embodimentcorrespond to the controller 105, the image adjustor 110, and the server106 of the first embodiment, respectively. The image processor 411, theimage generator 412, and the second communicator 413 of the fourthembodiment correspond to the image processor 111, the image generator112, and the second communicator 113 of the first embodiment,respectively.

FIG. 14 illustrates two cameras 402, two first communicators 403, twoimage displays 404, two controller 405, and two image adjustors 410,respectively, for the sake of explanation. The cameras 402, the firstcommunicators 403, the image displays 404, the controllers 405, and theimage adjustors 410 may be three or more. The image display 404, thecontroller 405, and the image adjustor 410 used by a first user USa area first image display 404 a, a first controller 405 a, and a first imageadjustor 410 a. The image display 404, the controller 405, and the imageadjustor 410 used by a second user USb are a second image display 404 b,a second controller 405 b, and a second image adjustor 410 b.

In the second image adjustor 410 a, the image processor 411, the imagegenerator 412, and the second communicator 413arean image processor 411a, an image generator 412 a, and a second communicator 413 a. In thesecond image adjustor 410 b, the image processor 411, the imagegenerator 412, and the second communicator 413are an image processor 411b, an image generator 412 b, and a second communicator 413 b. The firstimage adjustor 410 a is connected to a network via the secondcommunicator 413 a of the first image adjustor 410 a. The second imageadjustor 410 b is connected to the network via the second communicator413 b of the second image adjustor 410 b.

One of the two cameras 402 is a first camera 4021 and the other is asecond camera 4022. One of the two first communicators 403 is a firstcommunicator 4031 and the other is a first communicator 4032. The firstcamera 4021 is connected to the network via the first communicator 4031,and the second camera 4022 is connected to the network via the firstcommunicator 4032. The first camera 4021 and the second camera 4022 maybe connected to the network commonly via the first communicator 4031 orthe first communicator 4032. The server 406 is connected to the network.

The first camera 4021 and the second camera 4022 can move independentlyof each other. The first camera 4021 and the second camera 4022 have aGPS function. The server 406 can continuously acquire positions (currentpositions) of the first camera 4021 and the second camera 4022 throughthe network, the one first communicator 4031 and the other firstcommunicator 4032. The first image adjustor 410 a can acquire thepositions of the first camera 4021 and the second camera 4022 from theserver 406 through the second communicator 413 a of the first imageadjustor 410 a and the network. The second image adjustor 410 b canacquire the positions of the first camera 4021 and the second camera4022 from the server 406 through the second communicator 413 b of thesecond image adjustor 410 b and the network.

The first image adjustor 410 a can acquire a shooting image IM41 imagedby the first camera 4021 through the one first communicator 4031, thesecond communicator 413 a of the first image adjustor 410 a, and thenetwork. The second image adjustor 410 b can acquire the shooting imageIM41 imaged by the first camera 4021 through the one first communicator4031, the second communicator 413 b of the second image adjustor 410 b,and the network.

The first image adjustor 410 a can acquire a shooting image IM42 imagedby the second camera 4022 through the other first communicator 4032, thesecond communicator 413 a of the first image adjustor 410 a, and thenetwork. The second image adjustor 410 b can acquire a shooting imageIM42 imaged by the second camera 4022 through the other firstcommunicator 4032, the second communicator 413 b of the second imageadjustor 410 b and the network. FIG. 14 illustrates a case where thefirst image adjustor 410 a acquires a shooting image IM41 imaged by thefirst camera 4021, and the second image adjustor 410 b acquires ashooting image IM42 imaged by the second camera 4022.

The server 406 may acquire the shooting image IM41 imaged by the firstcamera 4021 and the shooting image IM42 imaged by the second camera 4022via the one first communicator 4031, the other first communicator 4032,and the network. In this case, the first image adjustor 410 a acquiresthe shooting image IM41 imaged by the first camera 4021 and the shootingimage IM42 imaged by the second camera 4022 from the server 406 via thenetwork and the second communicator 413 a of the first image adjustor410 a. The second image adjustor 410 b acquires the shooting image IM41imaged by the first camera 4021 and the shooting image IM42 imaged by asecond camera 4022 from the server 406 through the network and thesecond communicator 413 b of the second image adjustor 410 b.

The server 406 can identify the first user USa or the first imagedisplay 404 a by the first user USa performing a login operation or thefirst image display 404 a being connected to the network through thesecond communicator 413 a. The server 406 can specify the second userUSb or the second image display 404 b by the second user USb performinga login operation or the second image display 404 b being connected tothe network through the second communicator 413 b.

The server 406 can recognize which shooting image is displayed on whichimage display. For example, as illustrated in FIG. 14, when the shootingimage IM41 imaged by the first camera 4021 is displayed on the firstimage display 404 a and the shooting image IM42 imaged by the secondcamera 4022 is displayed on the second image display 404 b, the server406 recognizes that the shooting image IM41 imaged by the first camera4021 is displayed on the first image display 404 a and the shootingimage IM42 imaged by the second camera 4022 is displayed on the secondimage display 404 b. In other words, the server 406 can recognize thatthe first user USa is watching the shooting image IM41 imaged by thefirst camera 4021, and can recognize that the second user USb iswatching the shooting image IM42 imaged by the second camera 4022.

The first image adjustor 410 a can recognize, from the server 406, thecurrent position of the first camera 4021 imaging the shooting imageIM41 and the second camera 4022 imaging the shooting image IM42, andthat the second image adjustor 410 b has acquired the shooting imageIM42 imaged by the second camera 4022. The second image adjustor 410 bcan recognize, from the server 406, the current position of the firstcamera 4021 imaging the shooting image IM41 and the second camera 4022imaging the shooting image IM42, and that the first image adjustor 410 ahas acquired the shooting image IM41 imaged by the first camera 4021.

The shooting image IM41 (IM42) acquired by the first image adjustor 410a is input to the image processor 411 a. The image processor 411 aanalyzes the shooting image IM41 (IM42) to recognize the horizontaldirection of the shooting image IM41 (IM42). The image processor 411 amay analyze the shooting image IM41 (IM42) to recognize the verticaldirection of the shooting image 1141 (IM42), or to recognize thehorizontal direction and the vertical direction of the shooting imageIM41 (IM42).

The first image adjustor 410 a corrects distortion of the shooting imageIM41 (IM42), executes image processing such as adjustment of thehorizontal position of the shooting image IM41 (IM42), and outputs theimage-processed shooting image IM41 (IM42) to the first image display404 a.

The shooting image IM42 (IM41) acquired by the second image adjustor 410b is input to the image processor 411 b. The image processor 411 banalyzes the shooting image IM42 (IM41) to recognize the horizontaldirection of the shooting image IM42 (IM41). The image processor 411 bmay analyze the shooting image IM42 (IM41) to recognize the verticaldirection of the shooting image IM42 (IM41), or to recognize thehorizontal direction and the vertical direction of the shooting imageIM42 (IM41).

The second image adjustor 410 b corrects distortion of the shootingimage IM42 (IM41), executes image processing such as adjustment of thehorizontal position of the shooting image IM42 (IM41), and outputs theimage-processed shooting image IM42 (IM41) to the second image display404 b.

The server 406 may correct distortion of the shooting image 1M41 imagedby the first camera 4021 and the shooting image IM42 imaged by thesecond camera 4022, executes image processing such as adjusting thehorizontal position of the shooting image IM41 imaged by the firstcamera 4021 and the shooting image IM42 imaged by the second camera4022, and output the processed shooting image IM41 and shooting imageIM42 to the first image adjustor 410 a or to the second image adjustor410 b.

The first image display 404 a displays the shooting image IM41 (IM42)subjected to image processing by the first image adjustor 410 a or theserver 406. By mounting the first image display 404 a on the head of thefirst user USa, the first user USa can watch the shooting image IM41(IM42) subjected to image processing by the first image adjustor 410 aor the server 406.

The second image display 404 b displays a shooting image IM42 (IM41)subjected to image processing by the second image adjustor 410 b or theserver 406. By mounting the second image display 404 b on the head ofthe second user USb, the second user USb can watch the shooting imageIM42 (IM41) subjected to image processing by the second image adjustor410 b or the server 406.

While the image display 404 a (404 b) is mounted on the head of the userUSa (USb), the image display 404 a (404 b) generates attitudeinformation PN4 a (PN4 b) based on the direction in which the user USa(USb) faces and the attitude or the like of the user USa (USb). Theimage processor 411 a (411 b) acquires the attitude information PN4 a(PN4 b) from the image display 404 a (404 b). That is, the imageprocessor 411 a (411 b) acquires the attitude information PN4 a (PN4 b)based on the attitude of the image display 404 a (04 b).

Based on the attitude information PN4 a, the image processor 411 a ofthe first image adjustor 410 a displays, on the first image display 404a, an image of a region corresponding to a direction in which the firstuser USa faces and a state such as the attitude of the first user USafrom the shooting image IM41 imaged by the first camera 4021 or theshooting image IM42 imaged by the second camera 4022. Based on theattitude information PN4 b, the image processor 411 b of the secondimage adjustor 410 b displays, on the second image display 404 b, animage of a region corresponding to a direction in which the second userUSb faces and a state such as the attitude of the second user USb fromthe shooting image IM41 imaged by the first camera 4021 or the shootingimage IM42 imaged by the second camera 4022.

The controller 405 a (405 b), while being attached to the hand of theuser USa (USb), generates instruction information NN4 a (NN4 b) based onthe state of the hand movement or attitude of the user USa (USb). Theimage processor 411 a (411 b) acquires the instruction information NN4 a(NN4 b) from the controller 405 a (405 b).

Based on the instruction information NN4 a, the image processor 411 a ofthe first image adjustor 410 a can change or adjust the shooting imageIM41 or the shooting image IM42 displayed on the first image display 404a. Based on the instruction information NN4 b, the image processor 411 bof the second image adjustor 410 b can change or adjust the shootingimage IM41 or the shooting image IM42 displayed on the second imagedisplay 404 b.

The image generator 412 a of the first image adjustor 410 a generates acomposite image CM4 b which is a CG to be synthesized with the shootingimage IM41 (IM42). The image generator 412 b of the second imageadjustor 410 b generates a composite image CM4 a which is a CG to besynthesized with the shooting image IM42 (IM41). The composite image CM4a (CM4 b) is an image of a character such as an avatar. The compositeimage CM4 a is the avatar of the first user USa, and the composite imageCM4 b is the avatar of the second user USb.

When the first user USa operates the first controller 405 a, the imagegenerator 412 a of the first image adjustor 410 a generates thecomposite image CM4 b which is the avatar of the second user USb andstores it in a built-in memory or an external memory. When the seconduser USb operates the second controller 405 b, the image generator 412 bof the second image adjustor 410 b generates the composite image CM4 awhich is the avatar of the first user USa and stores it in the built-inmemory or the external memory.

The image processor 411 a (411 b) may acquire the composite image CM4 b(CM4 a) generated by the image generator 412 a (412 b), and output thecomposite image CM4 b (CM4 a) to the server 406 via the secondcommunicator 413 a (413 b) and the network.

When the user USa (USb) operates the controller 405 a (405 b), the imagegenerator 412 a (412 b) may generate the composite image CM4 a (CM4 b)that is the avatar of the user USa (USb). The image processor 411 a (411b) may acquire the composite image CM4 a (CM4b) generated by the imagegenerator 412 a (412 b) and output it to the server 406 via the secondcommunicator 413 a (413 b) and the network. That is, the first user USamay set the avatar of the second user USb or may set its own avatar. Thesecond user USb may set the avatar of the first user USa or may set itsown avatar.

The server 406 stores the composite image CM4 a (CM4 b) in a built-inmemory or an external memory in association with the user USa (USb). Theserver 406 may automatically set the composite image CM4 a (CM4 b) inassociation with the user USa (USb) and store it in the built-in memoryor the external memory. Based on the instruction information NN4 a (NN4b), the image processor 411 a (411 b) can acquire the composite imageCM4 b (CM4 a) from the image generator 412 a (412 b) or the server 406and display it on the image display 404 a (404 b).

The image generator 412 a (412 b) generates a spherical image VSS4 a(VSS4 b) that is a virtual image formed of a spherical surface that isCG. The image generator 412 a (412 b) stores the spherical image VSS4 a(VSS4 b) in a built-in memory or an external memory. Based on theinstruction information NN4 a (NN4 b), the image processor 411 a (411 b)acquires the spherical image VSS4 a (VSS4 b) generated by the imagegenerator 412 a (412 b) and displays it on the image display 404 a (404b).

The spherical image VSS4 a (VSS4 b) of the fourth embodiment correspondsto the spherical image VSS1 of the first embodiment. The user USa (USb)moves the hand on which the controller 405 a (405 b) is attached to aposition corresponding to the spherical image VSS4 a (VSS4 b) displayedon the image display 404 a (404 b), and feels as if the hand of the userUSa (USb) is in contact with the spherical image VSS4 a (VSS4 b).

The controller 405 a (405 b) may have an actuator disposed in a portionof the user USa (USb) in contact with the hand. Based on the instructioninformation NN4 a (NN4 b), the image processor 411 a (411 b) operatesthe actuator when it is determined that the hand of the user USa (USb)has moved to a position corresponding to the spherical image VSS4 a(VSS4 b). When the actuator applies pressure to the hand of the user USa(USb), the user USa (USb) can feel the sensation that the hand contactsthe spherical image VSS4 a (VSS4 b).

In a state where the spherical image VSS4 a is displayed on the firstimage display 404 a, when the first user USa moves the hand on which thefirst controller 405 a is attached in an arbitrary direction, the imageprocessor 411 a of the first image adjustor 410 a performs imageprocessing based on the instruction information NN4 a so that thespherical image VSS4 a and the shooting image IM41 or the shooting imageIM42 displayed on the first image display 404 a move in accordance withthe moving direction, moving speed and position of the hand of the firstuser USa.

In a state where the spherical image VSS4 b is displayed on the secondimage display 404 b, when the second user USb moves the hand on whichthe second controller 405 b is attached in an arbitrary direction, theimage processor 411 b of the second image adjustor 410 b, based on theinstruction information NN4 b, performs image processing so that thespherical image VSS4 b displayed on the second image display 404 b andthe shooting image IM41 or the shooting image IM42 move in accordancewith the moving direction, moving speed and position of the hand of thesecond user USb.

The user USa (USb) can rotate the spherical image VSS4 a (VSS4 b) in anarbitrary direction to an arbitrary position at an arbitrary speed bymoving the hand in an arbitrary direction to an arbitrary position at anarbitrary speed. That is, the user USa (USb) can rotate the sphericalimage VSS4 a (VSS4 b) by the hand movement. The image processor 411 a(411 b) moves the shooting image IM41 (IM42) in accordance with therotation of the spherical image VSS4 a (VSS4 b).

The image processor 411 a (411 b) can determine to which position on thecoordinate of the zenith ZE of the spherical image VSS4 a (VSS4 b)before the user USa (USb) rotates the spherical image VSS4 a (VSS4 b)has moved by the user USa (USb) rotating the spherical image VSS4 a(VSS4 b).

The image processor 411 a (411 b) calculates, based on the movingdirection of the zenith ZE and the position of the moving destination onthe coordinate of the spherical image VSS4 a (VSS4 b), the amount ofchange in the spherical image VSS4 a (VSS4 b) between the time beforeand after the user USa (USb) rotates the spherical image VSS4 a (VSS4 b)and the time after the user USa (USb) rotates the spherical image VSS4 a(VSS4 b). The change amount of the spherical image VSS4 a (VSS4 b) ofthe fourth embodiment corresponds to the change amount of the sphericalimage VSS1 of the first embodiment.

The image processor 411 a (411 b) stores the amount of change in thespherical image VSS4 a (VSS4 b) as the correction value CV4 a (CV4 b).The correction value CV4 a (CV4 b) of the fourth embodiment correspondsto the correction value CV1 of the first embodiment. The image processor411 a (411 b) may store the coordinates on the spherical image VSS4 a(VSS4 b) of the zenith ZE after the user USa (USb) has rotated thespherical image VSS4 a (VSS4 b) as the correction value CV4 a (CV4 b).

The image processor 411 a (411 b) may store the correction value CV4 a(CV4 b) in a built-in memory or an external memory, or may output thecorrection value CV4 a (CV4 b) to the server 406 via the secondcommunicator 413 a (413 b) and the network. The server 406 stores thecorrection value CV4 a (CV4 b) in the built-in memory or the externalmemory in association with the image display 404 a (404 b) or the userUSa (USb).

The image adjustment system 401 can display the composite image CM4 b,which is the avatar of the second user USb, on the first image display404 a mounted on the first user USa, by synthesizing it with theshooting image IM41 (IM42). The image adjustment system 401 can displaythe composite image CM4 a, which is the avatar of the first user USa, onthe second image display 404 b mounted on the second user USb bysynthesized it with the shooting image IM42 (IM41).

With reference to the flowcharts illustrated in FIGS. 15 and 16, anexample of an image adjustment method according to the fourth embodimentwill be described. More specifically, an example of a method ofsynthesizing the shooting image IM41 with the composite image CM4 b willbe described. The first image display 404 a is mounted on the head ofthe first user USa, and the first controller 405 a is attached on thehand of the first user USa. The second image display 404 b is mounted onthe head of the second user USb, and the second controller 405 b isattached on the hand of the second user USb. The first image display 404a displays the shooting image IM41 imaged by the first camera 4021, andthe second image display 404 b displays the shooting image IM42 imagedby the second camera 4022.

If the user USa (USb) determines that the shooting image IM41 (IM42)displayed on the image display 404 a (404 b) is not horizontal, in FIG.15, the user USa (USb) operates the controller 405 a (405 b), so thatthe image processor 411 a (411 b) causes the image display 404 a (404 b)to display a setting screen in step S401.

When the user USa (USb) operates the controller 405 a (405 b) to selecta predetermined item (e.g. horizontal adjustment item) displayed on thesetting screen, the image processor 411 a (411 b) shifts the processingto a predetermined processing mode corresponding to the selected item instep S402. When the horizontal adjustment item is selected, the imageprocessor 411 a (411 b) shifts the processing to a processing mode foradjusting the horizontal of the shooting image 1M41 (IM42) (horizontaladjustment mode).

In step S403, the image processor 411 a (411 b) acquires the sphericalimage VSS4 a (VSS4 b) generated by the image generator 412 a (412 b) anddisplays it on the image display 404 a (404 b). The image display 404 a(404 b) displays a mixture of the shooting image IM41 (IM42) and thespherical image VSS4 a (VSS4 b).

The user USa (USb) rotates the spherical image VSS4 a (VSS4 b) so thatthe shooting image IM41 (IM42) is horizontal, and the image processor411 a (411 b) moves the shooting image IM41 (IM42) displayed on theimage display 404 a (404 b) in step S404 in correspondence with therotation of the spherical image VSS4 a (VSS4 b). The user USa (USb) mayrotate the spherical image VSS4 a (VSS4 b) a plurality of times untilthe shooting image IM41 (IM42) becomes horizontal.

When the user USa (USb) determines that the shooting image IM41 (IM42)has become horizontal, the controller 405 a (405 b) is operated so thatthe image processor 411 a (411 b) terminates the display of thespherical image VSS4 a (VSS4 b) in step S405, and causes the imagedisplay 404 a (404 b) to display the setting screen. By leveling theshooting image IM41 (IM42) displayed on the image display 404 a (404 b),the zenith of the camera 4021 (4022) and that of the user USa (USb) canbe matched.

When the user USa (USb) operates the controller 405 a (405 b) to selecta predetermined item (e.g. end item) displayed on the setting screen,the image processor 411 a (411 b) shifts the processing to apredetermined processing mode corresponding to the selected item in stepS406. When the item to be ended is selected, the image processor 411 a(411 b) shifts the processing to a processing mode (exit mode) forending the horizontal adjustment.

In step S407, the image processor 411 a (411 b) acquires the amount ofrotation before and after the rotation (rotation angle) of the sphericalimage VSS4 a (VSS4b) as the amount of change of the spherical image VSS4a (VSS4 b). In step S408, the image processor 411 a (411b) stores theamount of change in the spherical image VSS4 a (VSS4 b) as thecorrection value CV4 a (CV4 b), and ends the process.

With reference to a flowchart illustrated in FIG. 16, a case will bedescribed in which the composite image CM4 b, which is the avatar of thesecond user USb, is displayed on the first image display 404 a watchedby the first user USa, while being synthesized with the shooting imageIM41. In FIG. 16, in step S411, the image processor 411 a identifies thefirst camera 41 that is imaging the obtained shooting image IM4021.

In step S412, the image processor 411 a acquires, from the first camera4021 or the server 406, the positional information of the first camera4021. The position information of the first camera 4021 includesinformation on the current position of the first camera 4021 and thedirection in which the first camera 4021 faces or moves.

In step S413, the image processor 411 a acquires information on othercameras (cameras other than the first camera 4021) from the server 406.For example, in step S413, the image processor 411 a acquiresinformation on the second camera 4022 from the server 406. Theinformation of the second camera 4022 includes the information of thesecond user USb watching the shooting image IM42 imaged by the secondcamera 4022, the present position of the second camera 4022, and thedirection in which the second camera 4022 faces or moves. The directionin which the second camera 4022 faces or the direction in which thesecond camera moves corresponds to the direction in which the seconduser USb faces, and can be specified based on the shooting image IM42displayed on the second image display 404 b.

In step S414, the image processor 411 a determines whether or notanother camera (cameras other than the first camera 4021) is imaged inthe shooting image IM41, that is, whether or not another camera iscaptured, for example, by image recognition. If it is determined in stepS414 that no other camera is imaged in the shooting image IM41 (NO), theimage processor 411 a returns the process to step S414.

If it is determined in step S414 that another camera is imaged in theshooting image IM41 (YES), then in step S415, the image processor 411 aidentifies the other camera imaged in the shooting image IM41 based onthe information of the other camera acquired in step S413.

FIG. 17 illustrates a state in which the first camera 4021 is positionedat a point A and the second camera 4022 is positioned at a point B infront of the first camera 4021. In the state illustrated in FIG. 17, theshooting image IM41 of the first camera 4021 includes another camera,that is, the second camera 4022. The image processor 411 a specifies,based on the information of the second camera 4022 acquired in stepS413, that the camera imaged in the shooting image IM41 of the firstcamera 4021 is the second camera 4022.

In step S416, the image processor 411 a specifies, based on the positioninformation of the first camera 4021 acquired in step S412 and theinformation of the second camera 4022 acquired in step S413, theposition of the second camera 4022 with respect to the first camera 4021and the direction in which the second camera 4022 faces or moves.

In step S417, the image processor 411 a identifies the user US who iswatching the shooting image IM imaged by the camera identified in stepS415. For example, the image processor 411 a specifies the second userUSb watching the shooting image IM42 imaged by the second camera 4022.In step S418, the image processor 411 a acquires the composite image CM4b, which is the avatar of the second user USb. When the composite imageCM4 b is stored in the server 406, the image processor 411 a acquiresthe composite image CM4 b from the server 406.

In step S419, the image processor 411 a synthesizes the composite imageCM4 b with the position of the shooting image IM41 corresponding to thesecond camera 4022, and displays the composite image CM4 b on the firstimage display 404 a. In steps S401 to S408, when the shooting image IM41is horizontally adjusted, the image processor 411 a horizontally adjuststhe composite image CM4 b in the same manner as the shooting image IM41and displays it on the first image display 404 a. When the correctionvalue CV4 a is stored, the image processor 411 a horizontally adjuststhe composite image CM4 b based on the correction value CV4 a, anddisplays the result on the first image display 404 a.

The image processor 411 a may change the size of the composite image CM4b in accordance with the position of the second camera 4021 with respectto the first camera 4022. For example, when the camera 4022 is locatedfar away from the camera 4021, the image processor 411 a performs imageprocessing on the composite image CM4 b so that the size of thecomposite image CM4 b becomes smaller in accordance with the distancebetween the first camera 4021 and the second camera 4022, andsynthesizes the composite image CM4 b with the shooting image IM41.

The image processor 411 a may change the direction of the compositeimage CM4 b in accordance with the direction in which the second camera4022 is facing or moving. The image processor 411 a may acquire acomposite image CM4 b having a direction corresponding to the directionin which the second camera 4022 is directed or the direction in which itis moving.

In the image adjustment system 401, the image adjustor 410 a (410 b),and the image adjustment method according to the fourth embodiment, theimage display 404 a (404 b) displays the spherical image VSS4 a (VSS4b). With the image adjustment system 401, the image adjustor 410 a (410b), and the image adjustment method according to the fourth embodiment,when the horizontal direction of the shooting image IM41 (IM42) iserroneously detected, or when the horizontal direction or thedesignation of the zenith ZE is deviated, the user USa (USb) can operatethe controller 405 a (405 b) to rotate the spherical image VSS4 a (VSS4b), thereby adjusting the shooting image IM41 (IM42) displayed on theimage display 404 a (404 b) so as to be horizontal.

Therefore, with the image adjustment system 401, the image adjustor 410a (410 b), and the image adjustment method according to the fourthembodiment, when the horizontal direction of the shooting image IM41(IM42) is erroneously detected or the horizontal or zenith designationis shifted, the user US can easily correct the horizontal or zenith ofthe shooting image IM41 (IM42).

With the image adjustment system 401, the image adjustor 410 a (410 b),and the image adjustment method according to the fourth embodiment, whenthe correction value CV4 a (CV4 b) is stored, the image processor 411 a(411 b) reads the correction value CV4 a (CV4 b), adjusts the image ofthe shooting image IM41 (IM42) taken by the camera 4021 (4022) based onthe correction value CV4 a (CV4 b), and displays the adjusted image onthe image display 404 a (404 b).

In the image adjustment system 401, the image adjustor 410 a (410 b),and the image adjustment method according to the fourth embodiment, theposition of the first camera 4021 relative to the second camera 4022 isspecified, and the position of the second camera 4022 in the shootingimage IM is synthesized with a composite image CM4 b which is the avatarof the second user USb watching the shooting image IM4022 imaged by thesecond camera 42. Thus, the first user USa can obtain a sense of sharingthe same space as the second user USb.

The present invention is not limited to the above-described embodiments,and various modifications can be made without departing from the gist ofthe present invention.

For example, the virtual image VSS generated by the CG may be anellipsoidal surface (ellipsoid), and may be any closed surface (closedsurface) within the reach of the user US. That is, it is only necessaryto obtain a feeling that the user US comes into contact with the closedcurved surface from the inside, so that the horizontal adjustment of theshooting image IM can be performed. Since the user US rotates thevirtual image VSS, the virtual image VSS preferably has a sphericalshape close to a spherical surface such as a spherical surface or anelliptical surface.

What is claimed is:
 1. An image adjustment system, comprising: a camera; an image adjustor configured to adjust a shooting image imaged by the camera; an image display configured to display the shooting image adjusted by the image adjustor; and a controller configured to output instruction information to the image adjustor, wherein the image adjustor comprises an image generator configured to generate a spherical image and a composite image to be synthesized with the shooting image, and an image processor configured to acquire the spherical image generated by the image generator based on the instruction information and display the spherical image on the image display, rotate the spherical image based on the instruction information, adjust the shooting image displayed on the image display in accordance with rotating the spherical image, adjust the composite image in accordance with the adjusted shooting image, and synthesize the adjusted composite image with the adjusted shooting image, the camera is an omnidirectional camera configured to image a range of 360 degrees, the image display is a head-mounted display capable of mounting on a head of a user, the controller is a glove type controller capable of attaching on a hand of the user, and when the user watches the spherical image displayed on the image display in a state where the image display is mounted on the head of the user, the spherical image is a virtual image arranged around the user and the image display and set to display within a range where the hand or a finger of the user is reached to the spherical image.
 2. The image adjustment system of claim 1, wherein when the user moves the hand or finger of the user in an arbitrary direction while the controller is attached on the hand of the user, the image processor rotates the spherical image corresponding to the movement of the hand or finger of the user based on the instruction information, and adjusts the shooting image displayed on the image display corresponding to the rotation of the spherical image.
 3. The image adjustment system of claim 2, wherein The image processor acquires an amount of changing between before and after the spherical image is rotated, and adjusts the shooting image and the composite image based on the amount of changing.
 4. The image adjustment system of claim 1, wherein the image processor adjusts horizontal positions of the shooting image and the composite image displayed on the image display in accordance with rotating the spherical image.
 5. The image adjustment system of claim 1, wherein the image processor separates a horizontal plane of the shooting image from a horizontal plane of the composite image, rotates the spherical image based on the instruction information, adjusts the shooting image displayed on the image display in accordance with rotating the spherical image, adjusts the composite image in accordance with the adjusted shooting image, and synthesizes the adjusted composite image with the adjusted shooting image.
 6. The image adjustment system of claim 5, wherein the image processor acquires an amount of changing between before and after the spherical image is rotated, and adjusts the shooting image based on the amount of changing.
 7. The image adjustment system of claim 6, wherein the image processor adjusts the composite image by matching the horizontal plane of the composite image with the horizontal plane of the shooting image based on the amount of changing.
 8. The image adjustment system of claim 1, wherein: the image adjustor includes a plurality of image adjustors each configured to adjust a shooting image imaged by the camera; the image display includes a plurality of image displays each configured to display the shooting image adjusted by a corresponding image adjustor of the plurality of image adjustors corresponding to a plurality of users; the controller includes a plurality of controllers each configured to output instruction information to a corresponding image display of the plurality of image displays corresponding to the plurality of users; and the image adjustment system further comprises a server configured to specify a first user of the plurality of users or a first image display of the plurality of image displays.
 9. The image adjustment system of claim 1, wherein: the camera includes a plurality of cameras; the image adjustor is configured to adjust a shooting image imaged by a first camera of the plurality of cameras; and the image adjustor comprises the image processor configured to, when a second camera of the plurality of cameras is imaged in the shooting image, synthesize the adjusted composite image in accordance with position information of the first camera and position information of the second camera with the shooting image in accordance with the position information of the second camera.
 10. An image adjustor, comprising: an image generator configured to generate a spherical image and a composite image to be synthesized with a shooting image imaged by a camera and displayed on a image display, and an image processor configured to acquire the spherical image generated by the image generator based on instruction information acquired from a controller and display the spherical image on the image display, rotate the spherical image based on the instruction information, adjust the shooting image displayed on the image display in accordance with rotating the spherical image, adjust the composite image in accordance with the adjusted shooting image, and synthesize the adjusted composite image with the adjusted shooting image, wherein the image display is a head-mounted display capable of mounting on a head of a user, and the controller is a glove type controller capable of attaching to a hand of the user.
 11. An image adjustment method, comprising: acquiring instruction information from a controller by a image processor; acquiring a spherical image by the image processor, the spherical image generated by an image generator based on the instruction information; displaying the spherical image by an image display; rotating the spherical image by the image processor based on the instruction information; adjusting a shooting image imaged by a camera and displayed on the image display in accordance with rotating the spherical image by the image processor, acquiring a composite image by the image processor, the composite image generated by the image generator and to be synthesized with the shooting image displayed on the image display; adjusting the composite image in accordance with the adjusted shooting image and synthesizing the adjusted composite image with the adjusted shooting image, by the image processor, wherein the image display is a head-mounted display capable of mounting on a head of a user, and the controller is a glove type controller capable of attaching to a hand of the user. 